fix: prevent stale prompt cache session overwrites#372
Conversation
Overlapping requests that share a prompt cache session can finish out of order. Older responses were allowed to overwrite newer session-affinity state, which lets stale previous_response_id and account affinity win. Version session-affinity writes per request so the newest in-flight request for a session keeps control of the stored response id and preferred account.
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 19 minutes and 51 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Superseded by #387, which rebuilds the full open PR stack onto one reviewed integration branch. |
|
Closing in favor of #387. |
Problem
Overlapping requests that share a prompt-cache session can finish out of order.
Before this change, older responses were allowed to overwrite newer session-affinity state, which meant stale
previous_response_idvalues and stale preferred-account affinity could win after a newer request had already completed.Fix
Version session-affinity writes per request and reject stale writes.
This ensures the newest in-flight request for a session keeps control of both:
previous_response_idChanges
lib/session-affinity.tswriteVersionrememberWithVersion(...)rememberWithVersion(...)andupdateLastResponseId(...)callsindex.tstest/session-affinity.test.tstest/index.test.tsValidation
note: greptile review for oc-chatgpt-multi-auth. cite files like
lib/foo.ts:123. confirm regression tests + windows concurrency/token redaction coverage.Greptile Summary
this PR fixes stale prompt-cache session overwrites by adding a per-request monotonic
writeVersionto session-affinity entries and rejecting writes whose version is older than the stored one. the core streaming mechanism is sound and the new regression test validates the primary overlapping-stream scenario.rememberWithVersionandupdateLastResponseIdboth guard onentry.writeVersion > writeVersion, preventing out-of-order responses from clobbering newer statesessionAffinityWriteVersionat module scope keeps the counter monotonic across plugin config reloads — a reset to0would break stale-write preventiontest/index.test.tsoverlapping-stream test fires callbacks out-of-order and asserts the newest account andprevious_response_idwinremember()passesDate.now()(~1.7e12) aswriteVersion, whileonResponseIduses integer versions (1, 2, 3 …). for non-streaming requests whereresponseContinuationEnabled=trueandonResponseIdnever fires, the fallbackremember()on the success path writes a timestamp-scale version and permanently blocks all subsequent integer-versionedrememberWithVersioncalls for that sessionConfidence Score: 3/5
safe to merge for the streaming path; non-streaming requests with response continuation enabled can permanently freeze session affinity after one response-ID-less response
the primary overlapping-stream bug is correctly fixed and well-tested. however, the mixed integer/timestamp version space is an existing concurrency issue not fully addressed by this PR — one non-streaming response per affected session can permanently poison its versioning, and there is no test covering this path.
lib/session-affinity.ts and the fallback remember() call sites in index.ts (success path around line 2397) need attention for the version-space contamination fix
Important Files Changed
Sequence Diagram
sequenceDiagram participant R1 as Request 1 (v=1) participant R2 as Request 2 (v=2) participant SA as SessionAffinityStore participant R3 as Request 3 (v=3) R1->>SA: rememberWithVersion(key, acc0, t1, v=1) R2->>SA: rememberWithVersion(key, acc1, t2, v=2) Note over SA: writeVersion = 2 SA-->>R2: onResponseId(resp_second) → updateLastResponseId(v=2) ✓ SA-->>R1: onResponseId(resp_first) → updateLastResponseId(v=1) ✗ stale, rejected Note over SA: stored: acc1, resp_second, writeVersion=2 ✅ Note over R1: non-streaming fallback: remember(key, acc0) R1->>SA: rememberWithVersion(key, acc0, now, writeVersion=Date.now()≈1.7e12) Note over SA: writeVersion = 1_700_000_000_000 ⚠️ R3->>SA: rememberWithVersion(key, acc1, t3, v=3) Note over SA: 1.7e12 > 3 → REJECTED ❌ affinity frozenComments Outside Diff (1)
lib/session-affinity.ts, line 127-154 (link)writeVersionis used as two incompatible things:sessionAffinityVersion = 1, 2, 3 …) assigned in theonResponseIdcallbackDate.now() ≈ 1_700_000_000_000) assigned by theremember()fallback, which callsrememberWithVersion(key, idx, now, now)the stale guard
entry.writeVersion > writeVersion(lines 143-145 and 83-85) works correctly when both sides use the same version space — but once a timestamp-versioned entry is stored, every future integer-versioned write is rejected because1.7e12 >> 3.the contamination path exists today in
index.ts:this fires for non-streaming requests where
responseContinuationEnabled=truebutonResponseIdnever fires (storedResponseIdForSuccessstaysfalse). after that one call, all subsequentrememberWithVersion(…, sessionAffinityVersion)writes for that session are silently rejected, freezing the preferred account and response-id indefinitely.fix — thread the per-request integer version into the fallback success path in
index.ts:Prompt To Fix With AI
Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "fix: prevent stale prompt cache session ..." | Re-trigger Greptile